A semiconductor memory device having a three-dimensional structure has been proposed in which a memory hole is formed in a stacked body including a plurality of electrode layers stacked thereon, a columnar portion is provided in the memory hole, a plurality of charge storage portions are respectively provided between the plurality of electrode layers and the columnar portion in a stacking direction of the stacked body. In order to increase the capacity of the semiconductor memory device, it is desired to increase the number of stacked electrode layers. Increasing the number of stacked electrode layers makes it difficult to form a memory hole having a uniform diameter in the stacking direction. Variations in the diameter of the memory hole in the stacking direction can lead to variations in the coupling ratio between the electrode layer and the channel.